-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] Chunk distribution strategies #824
base: dev
Are you sure you want to change the base?
[WIP] Chunk distribution strategies #824
Conversation
This pull request introduces 1 alert when merging 96841f9 into b2664b7 - view on LGTM.com new alerts:
|
96841f9
to
6da1db3
Compare
This pull request introduces 1 alert when merging 6da1db3 into a6287fc - view on LGTM.com new alerts:
|
6da1db3
to
43b1e31
Compare
This pull request introduces 1 alert when merging 43b1e31 into f6b0054 - view on LGTM.com new alerts:
|
43b1e31
to
5f2b347
Compare
This pull request introduces 1 alert when merging 5f2b347 into f6b0054 - view on LGTM.com new alerts:
|
5f2b347
to
bb7fa45
Compare
This pull request introduces 1 alert when merging bb7fa45 into 9b349eb - view on LGTM.com new alerts:
|
8c7fc78
to
9fc6d6b
Compare
@ax3l ping |
9fc6d6b
to
a1bb1e0
Compare
This pull request introduces 1 alert when merging ebea6cf into 86a15ba - view on LGTM.com new alerts:
|
993cfb6
to
0fca8b7
Compare
I've added Python bindings now, including usage of one distribution strategy in openpmd-pipe as an integrated usage example. |
0fca8b7
to
ca6f0da
Compare
f9ba2a9
to
47653ac
Compare
47653ac
to
89fea78
Compare
One issue of this approach is that the attribute EDIT: I've transformed this to a parallel dataset now. Needs some more checking for char portability. Also, we should probably buffer these values on reading, since the table can only be read in steps in which it was written. In short: Make this less like an attribute that can be set and inquired, and rather more automatic during Series construction. |
62758f5
to
86d5a8c
Compare
ae23675
to
cc87760
Compare
37d0a0e
to
8b1a41d
Compare
166b3be
to
d734636
Compare
@@ -1044,6 +1131,53 @@ | |||
} | |||
#endif | |||
|
|||
#if openPMD_HAVE_MPI | |||
TEST_CASE("unavailable_backend", "[core][parallel]") |
Check notice
Code scanning / CodeQL
Unused static function Note test
autoRegistrar23
d734636
to
196d58e
Compare
196d58e
to
e63c9e1
Compare
e63c9e1
to
436360e
Compare
3c6550b
to
2f9ba73
Compare
2f9ba73
to
49b16f8
Compare
49b16f8
to
5e17686
Compare
5e17686
to
3745f58
Compare
3745f58
to
3c93be4
Compare
@todo Why do we need to increase the refcount twice??
3c93be4
to
60b78ad
Compare
60b78ad
to
bd227b7
Compare
Based on topic-available-chunks, see #802.Based on #1043
Adds strategies for guiding applications to efficient parallel chunk loading patterns.
Idea: PR #802 lets reading applications inquire the chunks that are available for loading in the backend. This PR adds a function
chunk_assignment::assignChunks
, taking such aChunkTable
as well as a strategy as input and produces a furtherChunkTable
that may be used as a load pattern.Implemented strategies:
n_ranks
hyperslabs of the data space.Also, add an attribute to the global
Series
object (currently calledrankMetaInfo
, I'm sure we'll change this) that can be used to write string-formatted meta information for the single writing processes and assign meaning to them (e.g. hostnames). Used by the hostname-based strategy.TODO: